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ABSTRACT 

In medical or health related problems, the phrase Receiver Operating Characteristic (ROC) curve plays a vital role 
to evaluate the assessment of classifier performance by its area under the curve (AUC). For binormal model, let the random 
variables (r.v’s) X and Y be denoted by diseased normal population (D) and healthy normal population (H) with means p x 
and p y and sample variances s x and s y respectively. In binormal model area under the ROC curve lies between 0 and 1. 
This paper presents, an AUC which is to be prefixed so called Target AUC, (for example 0.9, 0.8 etc) to attain such AUC, 
3c limits ( i.e. o, 2c and 3c) can be taken into the consideration to obtaining different AUCs basing on various possible 
distances (difference between population means) Aj . j = 1 , 2 ....9. Nine possible distances can be calculated from lower and 
upper limits of the confidence interval of means which are based on 3c limits and also comparisons made between them. 
New average methods namely i) Simple Average Method ii) Fixed Weights Method (FW-Method) and iii) Proportional 
Weights Method (PW -Method) are briefly discussed also significant comparisons made among the three methods. 

KEYWORDS: Comparison of AUCs of Binormal ROC Curve Using, Radar Images, X and Y be Denoted by Diseased 
Normal Population (D) and Healthy Normal Population (H) 

INTRODUCTION 

The phrase Receiver Operation Characteristic (ROC) analysis had its origin in Statistical Decision Theory as well 
as Signal Detection Theory (SDT) and was used during II World War for the analysis of radar images. To draw a ROC 
curve, only the true positive rate (TPR) and false positive rate (FPR) are needed (as functions of some classifier 
parameters). The TPR defines how many correct positive results occur among all positive samples available during the test. 
FPR, on the other hand, defines how many incorrect positive results occur among all negative samples available during the 
test. 

A ROC space is defined as TPR is plotted on the Y-axis and FPR is plotted on the X-axis. In other words, which 
depicts relative trade-offs between true positives and false positives Since TPR is equivalent with sensitivity and FPR is 
equal to 1 - specificity, the ROC graph is sometimes called the sensitivity verses (1 - specificity) plot. Each prediction 
result or instance of a confusion matrix represents one point in the ROC space. 

The four possible states of the confusion matrix, which are non-negative integers shown below. 



Table 1 



Test Result 


Diagnosis 


Positive 


Negative 


Positive 


TP 


FN 


Negative 


FP 


TN 
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The best possible prediction method would yield a point in the upper left corner or coordinate (0, 1) of the ROC 
space, representing 100% sensitivity (no false negatives) and 100% specificity (no false positives). The (0, 1) point is also 
called a perfect classification. In roc curve analysis, the classifier performance can often be explained by area under the roc 
curve (AUC). For a binary classifier it is shown that AUC = (TP+TN) / (TP+FN+FP+TN). A poor classifier will have 
AUC <= 0.50. 

Earlier I have proposed new AUC estimates of the binormal roc model, using 100(l-a) % confidence intervals and 
confidence interval of means basing on quartiles from 9 possible distances of two normal populations i.e. Diseased (D) and 
Healthy (H) populations with respective means and sample standard deviations. 

In this paper, I have been observed 3a limits (i.e. a, 2a and 3a) as confidence limits of the confidence interval of 
means basing on 9 possible distances to the estimation of new AUCs. New summarized auc estimates have been derived by 
Simple Average Method, Fixed Weights Method (FW -Method) and Proportional Weights Method (PW -Method) and made 
comparisons among them also normality was tested by P-P plot. 

AUC of the Binormal ROC Model 

Let X and Y be two continuous random variables representing the test variables of Diseased (D) and Healthy (H) 
groups respectively, such that X ~ N (p x , c x 2 ) and Y - N (p Y , a Y 2 ). 

Then the parametric form of the binormal ROC is given by 

ROC (t) = <D (a + b <D _1 (t)) (1) 

Where a = ( l ‘ x l '' Y ) and b =(— ) are constants. 

V °x / VOY/ 

Faraggi and Reiser (2002) have shown that AUC of the binormal ROC model, which is given by 




( 2 ) 



Let p x and p y represent the estimated means and sj, Sy represent the sample variances of X and Y respectively. 
The estimated AUC can be obtained by replacing the parameters with their sample estimates. This gives 



Mx-My 



Suppose, we wish to develop an ROC curve such that it has a predetermined AUC denoted by AUC*, like 0.9, 0.8 
etc. If Z* = cD' 1 (AUC*) will be denotes the standard normal deviate corresponding to AUC . Faraggi and Reiser (2002) 
have shown that the estimated mean of the Diseased (D) group is 




AUC = <D 



fix = fi y + Z*Js% + s£. (4) 

With the help of fi x fi y and s~, s?, the AUC can be estimated. 

Let A denotes the distance between the means of the two normal populations. Now A is defined as A = (fi x - fiy) 
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then the estimation of AUC can be reduced to the form of 



A 

J4+4 

The true population mean of the two populations can be anywhere in the confidence interval (Cl) the true 
difference between the means depending on the upper and lower limits of the intervals. In the following sections 3o limits 
can be treated as upper and lower limits of the confidence interval of means. 

The Possible Distances between the Means Basing on 3a Limits 

Usually the true mean of the population can be lie within the confidence interval. For each population there exist 
lower and upper limits of the confidence interval. For binormal distribution there exists 9 possible distances i.e. Aj; j=l, 
2... .9 which can be used to estimate new AUCs basing on 3a limits (i.e. a, 2a and 3a). Then equation (5) becomes 





Aj = Ol 






;j=i,2, 9 



(6) 



Lower and upper limits of p + a, p +2a and p + 3a limits are explained below. 

Case I: Consider p + a limits 

The Lower (L) and Upper (U) limits of p + a confidence interval are given by 
Mean + Standard Error of mean i.e. Mean + 

For diseased (D) normal population, the lower and upper limits of p + la are given by 

Mean -©- K ^ Mean -© le L 2 =K-{j=)^ K * P *-©= U2 

For healthy (H) normal population, the lower and upper limits of p + a are given by 

Mean - © - Py - Mean - S) i e L1= V © ^y ^ V © =U1 

Case II: Consider p + 2c limits 

The Lower (L) and Upper (U) limits of p + 2c confidence interval are given by 
Mean + 2 x Standard Error of mean i.e. Mean + 2 

For diseased (D) normal population, the lower and upper limits of p ±2c are given by 

Mean - 2 © 5 ^ Mean - 2 © Le - L2 = K- 2 (q=)^ K ^ K - 2 ©= U2 

For healthy (H) normal population, the lower and upper limits of p + 2c are given by 

Mean - 2 (S) - p y - Mean ■ 2 © Le ' L1= V 2 ©~ ? y - V 2 © =U1 

Case III: Consider p + 3a limits 
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The Lower (L) and Upper (U) limits of p + 3o confidence interval are given by 
Mean + 3 x Standard Error of mean i.e. Mean + 3 (JJ=) 

For diseased (D) normal population, the lower and upper limits of p + 3a are given by 

M “"- 3 (lf) s K - “ L 2 =fv 3 (|fL K - iv 3 (fr )= U2 

For healthy (H) normal population, the lower and upper limits of p + 3a are given by 

M “"- 3 (IrL K * M “" 3 (if) i=ri=(i y - 3 (g< p, < fv 3 t§)= ul 

The 9 possible true distances between the means basing on the limits are shown in Table 1. 



Table 2: List of Possible Distances between the Means 



A, 


a 2 


^3 


a 4 


A s 


Afi 


A? 


As 


a 9 


(L 2 -L,) 


(Lrp y ) 


(L 2 -U,j 


(fU-LD 


(Px-My) 


(K-Ui) 


(u 2 -l,) 


(U 2 -p v ) 


(U 2 -U,) 



Basing on these 9 possible distances different AUCs can be estimated for each case separately. New summarized 
AUC estimates can be obtained by three weighted average methods of all estimated AUCs are as given below. 

• Simple Average Method: Among three methods, it is the simplest one. The estimation of new AUC through this 

method is given by 

AUC AV g= Zy=i Wj Aj where Wj = 1/9 V j = 1, 2 9 

• Fixed Weights Method (FW-Method): In this method, AUC is defined as below 

AUC fw = Wj Aj +°-^A 5 where Wj = y V j + 5 

• Proportional Weights Method (PW-Method): In this method, AUC is given by the following formula 

YS-.Wj Ai i 

AUCpw = J ~ g where Wj = - V j = 1 , 2 9. 

tj =1 Wj J A j J 

Empirical Illustrations for Estimation of Binormal AUCs 

In binormal ROC model, the true difference between the means of two normal populations attains merely the 
target AUC=0.9. To obtain more than one target AUC, different AUCs (A,; j=l, 2 ...9) can be estimated basing on 9 
possible distances between the means of two normal populations i.e. Diseased (D) and Healthy (H) populations. These 9 
possible distances can be obtained from each case of 3o limits of the two normal populations such as p + o, p ±2a and p 
+ 3o limits. For each case, new summarized improved AUCs can be estimated by three average methods basing on weights 
i.e. £/= i Wj and to make comparisons among the three methods. Also normality was tested by P-P plot with respective R 2 
value. 

Illustration 1: Estimated AUC values and new summarized AUC estimates with normality fit by P-P plot from p 
+ c limits with R 2 value are shown in the following Table 2. 
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Table 3: Estimated AUC Values and Normality Fit by P-p Plot from p ± a Limits 



Target AUC = 0.9, p r = 85 


S. No 


Inputs 


Estimated AUC values 


Normality Fit by 
P-P Plot 


1 


n] = 5 ,n 2 = 5 
p x = 103.12 
S x = 10, Sy = 10 


{A,..A 5 } = {0.9000,0.8328,0.7419,0.9450,0.9000}, 
{A 6 ..A 9 } = {0.8328,0.9722,0.9450,0.9000} 
AUCavg = 0.8855, AUC FW = 0.8918 , AUC PW = 0.8626 


Y= 0.069X+0.885 
R 2 = 0.907 


2 


ni = 10, n 2 = 15 
p x = 103.12 
S X = 10, Sy = 10 


{A,...A 5 } ={0.8926,0.8550,0.8093,0.9284,0.9000} 

{ A 6 A 9 }= {0.8641,0.9543,0.9339,0.9070} 
AUCavg =0.8938, AUCpw =0.8965 , AUC PW =0.8855 


Y=0.044X+0.893 
R 2 = 0.964 



Illustration 2: Estimated AUC values and new summarized AUC estimates with normality fit by P-P plot from p 
+ 2o limits are shown in the following Table 3. 



Table 4: Estimated AUC Values and Normality Fit by P-P Plot from p ± 2a Limits 



Target AUC = 0.9, p r = 85 


S. No 


Inputs 


Estimated AUC values 


Normality Fit by 
P-P Plot 


1 


nj = 5 ,n 2 = 5 
p x = 103.12 
S x = 10, Sy = 10 


{A,..A 5 } = {0.9000,0., 0.7419,0. 5066,0. 9722,0. 9000}, 
{A 6 ..A 9 } = {0.7419,0.9946,0.9722,0.9000} 
AUCavg = 0.8477, AUC FW = 0.8706 , AUC PW = 0.5413 


Y= 0.145X+0.847 
R 2 = 0.829 


2 


n, = 10, n 2 = 15 
p x = 103.12 
S x = 10, Sy = 10 


{A,... A s } ={0.8848,0.7980,0.6805,0.9502,0.9000} 

{ A 6 A 9 }= {0.8203,0.9819,0.9581,0.9137} 
AUCavg =0.8764, AUC FW =0.8867 , AUC PW =0.8316 


Y=0.092X+0.876 
R 2 = 0.910 


3 


nj = 25, n 2 = 20 
p x = 103.12 
S x = 10, Sy = 10 


{A1...A4} ={0.9057,0.8410,0.7525,0.9450,0.9000} 
{A 6 ...A 9 }= {0.8328,0.9700,0.9411,0.8940} 
AUCavg = 0.8869, AUC FW = 0.8926 , AUC PW = 0.8668 


Y=0.066X+0.886 
R 2 = 0.929 



Illustration 3: Estimated AUC values and new summarized AUC estimates along with normality fit by P-P plot 
with R 2 value from p + 3o are shown in the following Table 4. 



Table 5: Estimated AUC Values and Normality Fit by P-P Plot from p ± 3a Limits 



S. No 


Inputs 


Estimated AUC values 


Normality Fit by 
P-P Plot 


1 


n] = 5 ,n 2 = 5 
p x = 103.12 
Sy = 10, Sy = 10 


{A,.. A s } = { 0.9000,0. ,0.6304,0.5000,0.987 1 ,0.9000 } , 
{A 6 ..A 9 } = {0.6304,0.9993,0.9871,0.9000} 
AUCavg = 0.8260, AUC FW = 0.8584 , AUC PW = 0.7420 


Y= 0.173X+0.826 
R 2 = 0.831 


2 


ni = 10, n 2 = 15 
p x = 103.12 
Sy = 10, Sy = 10 


{A 1 ...A 5 } ={0.8767,0.7293,0.52515,0.9663,0.9000} 

{ A 6 A 9 }= {0.7685,0.9938,0.9746,0.9199} 
AUCavg =0.8505, AUC FW =0.8721 , AUC PW =0.6222 


Y=0.142X+0.850 
R 2 = 0.859 


3 


ii| = 25, n 2 = 20 
p x = 103.12 
S x = 10, Sy = 10 


{ A, ... A 4 } = { 0.9085,0.8044,0.649 1 ,0.9604,0.9000 } 
{A 6 ...A 9 }= {0.7902,0.9854,0.9560,0.8909} 
AUCavg = 0.8717, AUCpw = 0.8841 , AUC PW = 0.81 12 


Y=0.101X+0.871 
R 2 = 0.888 



COmparision of New Summarized AUC Estimates 

After obtaining estimated AUC values basing on 9 possible distances from each case, new summarized AUC 
estimates can be computed by three average methods such as Simple Average Method, Fixed Weights Method (FW- 
Method) and Proportional Weights Method (PW-Method) and also significant comparisons made among the three methods 
are shown in the following diagrams. 
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Figure 1 



CONCLUSIONS 



At fixed mean of healthy normal population = 85 , diseased normal population mean has been estimated 
as / j x = 103.12. At both p x p Y and sample standard deviations S x , S y , different AUC estimates have been obtained using 
confidence interval of means basing on 9 possible distances from the confidence limits which can be obtained from 3a 
limits i.e. p + a, p + 2a and p + 3a limits. Among all three cases, in each case middle most possible distance used AUC 
would attain the Target AUC=0.9 and also observed that R 2 value has been extensively increased as sample size increases. 
In three cases, while considering the p + a limits, the estimated AUC values, new summarized AUC estimates and R 2 
values have been given an appropriate values and found that consideration of p + a limits would give best results among 
the three. Among three average methods, Fixed Weight Method provides better summarized AUC estimates other than two 
methods. And finally normality was tested by P-P plot which shows R 2 values are very close to 1 in easel. Moreover 
consideration of p + a limits is the best one as compared to p + 2a and p + 3a limits. 
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